[Inference Providers] Document automatic validation cron job#1822
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
Vaibhavs10
left a comment
Vaibhavs10
left a comment
There was a problem hiding this comment.
The validation part looks good! thanks for adding it! 🤗
|
care pushing just the part that changed? (sorry) |
|
git add > partial commit |
| For large language models (LLMs), additional behavioral tests are conducted: | ||
|
|
||
| - Tool calling support. | ||
| - Structured output support. |
There was a problem hiding this comment.
does this mark the model as in error as well? or it's a ≠ UI? (seems a bit heavy to mark it as Failing validation for this)
There was a problem hiding this comment.
It does not - for now it just feeds a map of booleans flagging wether or not the provider-llm pair supports the feature
The tests are only carried out if the initial test succeeds (first token received in < 5s)
It's not displayed nor used anywhere yet
8225a6e to
91dcdf7
Compare
4 participants
Document how we validate inference provider mappings
Sorry for all the whitespace changes 😬 My linter